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- The MAILING DATE of this communication appears on the cover sheet with the correspondence address- 

All claims being allowable, PROSECUTION ON THE MERITS IS (OR REMAINS) CLOSED in this application. If not included 
herewith (or previously mailed), a Notice of Allowance (PTOL-85) or other appropriate communication will be mailed in due course. THIS 
NOTICE OF ALLOWABILITY IS NOT A GRANT OF PATENT RIGHTS. This application is subject to withdrawal from issue at the Initiative 
of the Office or upon petition by the applicant. See 37 CFR 1.313 and MPEP 1308. 

1 . 13 This communication is responsive to 05/13/04 . 

2. S The allowed claim(s) is/are 21-15.20.22-23,28-35.37 and 39-42 . 

3. S The drawings filed on 30 October 2000 are accepted by the Examiner. 

4. □ Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-{d) or (f). 

a) □ All b) □ Some* c) □ None of the: 

1 . □ Certified copies of the priority documents have been received. 

2. □ Certified copies of the priority documents have been received in Application No. . 

3. □ Copies of the certified copies of the priority documents have been received in this national stage application from the 

International Bureau (PCT Rule 1 7.2(a)). 
* Certified copies not received: . 

Applicant has THREE MONTHS FROM THE "MAILING DATE" of this communication to file a reply complying with the requirements 
noted below. Failure to timely comply will result in ABANDONMENT of this application. 
THIS THREE-MONTH PERIOD IS NOT EXTENDABLE. 

5. □ A SUBSTITUTE OATH OR DECLARATION must be submitted. Note the attached EXAMINER'S AMENDMENT or NOTICE OF 

INFORMAL PATENT APPLICATION (PTO-152) which gives reason(s) why the oath or declaration is deficient. 

6. □ CORRECTED DRAWINGS ( as "replacement sheets") must be submitted. 

(a) □ including changes required by the Notice of Draftsperson's Patent Drawing Review ( PTO-948) attached 

1) □ hereto or 2) □ to Paper No./Mail Date _. 

(b) □ including changes required by the attached Examiner's Amendment / Comment or in the Office action of 

Paper No./Mail Date . 

Identifying indicia such as the application number (see 37 CFR 1.84(c)) should be written on the drawings in the front (not the back) of 
each sheet. Replacement sheet(s) should be labeled as such in the header according to 37 CFR 1.121(d). 

7. □ DEPOSIT OF and/or INFORMATION about the deposit of BIOLOGICAL MATERIAL must be submitted. Note the 

attached Examiner's comment regarding REQUIREMENT FOR THE DEPOSIT OF BIOLOGICAL MATERIAL. 
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5. □ Notice of Infomial Patent Application (PTO-152) 
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Paper No./Maii Date . 
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9. □ Other . 
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DETAILED ACTION 



1 . Claims 2-41 are In this application, claims 24-26 and 36 are canceled by the 
examiner amendment, claims 2-15, 20, 22-23, 28-35, 37 and 39-42 are pending in this 
application. 



2. An examiner's amendment to the record appears below. Should the changes 
and/or additions be unacceptable to applicant, an amendment may be filed as provided 
by 37 CFR 1.312. To ensure consideration of such an amendment, It MUST be 
submitted no later than the payment of the Issue fee. 

Authorization for this examiner's amendment was given in a telephone interview 
with Emmanuel Rivera on 05/13/04. 



Please delete claim [2], replace with "2. A method comprising: 
initiating a search for images based on at least one query keyword in a query; 
identifying, during the search, first images having associated keywords that match the 
query keyword, and second Images that contain low-level features similar to those of 
the first images identified as matching the query keyword, wherein the low-levels 
features comprise color, shape, and texture and the low-level features do not match the 
search query; 

ranking the first and second images; 

presenting the first and second Images to a user; and 



EXAMINER'S AMENDMENT 



Please cancel these claims 24-26 and 36. 
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receiving feedback from tfie user as to whether the first and second images are 
relevant to the query further comprising: 

assigning a large weight to an association between the query keyword and the 
images deemed relevant by the user and/or assigning a small weight to an association 
between the query keyword and the example image." 

Please delete claim [4], replace with "4. A method comprising: 

initiating a search for images based on at least one query keyword in a query; 

identifying, during the search, first images having associated keywords that 
match the query keyword, and second images that contain low-level features similar to 
those of the first images identified as matching the query keyword wherein the low- 
levels features comprise color, shape and texture and the low-level features do not 
match the search query; 

presenting the first and second images to a user; 

monitoring feedback from the user as to which of the first and second images are 
relevant to the query; and 

receiving feedback from the user as to whether the first and second images are 
relevant to the query comprising: 

assigning a large weight to an association between the query keyword and the 
images deemed relevant by the user and/or assigning a small weight to an association 
between the query keyword and the example image." 

Please cancel claim [5], replace with "5. A method comprising: 

initiating a search for images based on at least one query keyword in a query; 
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identifying, during the search, first images having associated keywords that 



match the query keyword, and second images that contain low-level features similar to 



those of the first images identified as matching the query keyword wherein the low- 
levels features comprise color, shape, and size and the low-level features do not match 
the search query; 

presenting the first and second images to a user; 

receiving feedback from the user as to whether the first and second images are 
relevant to the query; 

learning how the first and second images are identified based on the feedback 
from the user; and 

receiving feedback from the user as to whether the first and second images are 
relevant to the query further comprising: 

assigning a large weight to an association between the query keyword and the 
images deemed relevant by the user and/or assigning a small weight to an association 
between the query keyword and the example image." 

Please delete claim [12], replace with "12. A method comprising: 

permitting entry of both keyword-based queries and content-based queries; 
finding a first set of images using semantic-based image retrieval and a second set of 
images using low-level feature-based image retrieval based on the first set of images 
wherein the low-levels features comprise color, shape and texture and the low-level 
features do not match the semantic-based and the low-level features do not match the 
keyword-based or content-based; 
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presenting the images to a user so that the user can indicate whether the images 
are relevant; 

conducting semantic-based relevance feedback and low-level feature-based 
relevance feedback in an integrated fashion; and 

receiving feedback from the user as to whether the first and second images are 
relevant to the query further comprising: 

assigning a large weight to an association between the query keyword and the images 
deemed relevant by the user and/or assigning a small weight to an association between 
the query keyword and the example image." 

Please delete claim [20], replace with "20. A method comprising: 
identifying the set of images from the presenting retrieved using the at least one 
keyword and second images that contain low-level features similar to those of the first 
images identified as matching the at least one keyword, wherein the low-levels features 
comprise color, shape and texture and the low-level features do not match the search 
query; 

presenting a result set of images that are retumed from an image retrieval search 
of a query having at least one keyword; 

monitoring feedback from a user as to whether the images in the result set are 
relevant to the query; 

in an event that the user selects at least one image as being relevant to the 
query, associating the keyword in the query with the selected image to fonn a first 
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keyword-image association and assigning a comparatively large weight to the first 

keyword-image association; and 

in an event that the user identifies an example image for refinement of the 

search, associating the keyword in the query with the example image to form a second 

keyword-image association and assigning a comparatively small weight to the second 

keyword-image association." 

Please delete claim [28], replace with "28. An image retrieval system comprising: 
a query handler to handle both keyword-based queries having one or more 

search keywords and content-based queries having one or more low-level features of 

an image; and 

a feature and semantic matcher to identify at least one of (1) first images having 
keywords that match the search keywords from a keyword-based query, and (2) second 
images having low-level features similar to the low-level features of a content-based 
query based on the first images, wherein the feature and semantic matcher ranks the 
images, and the low-level features comprise color, shape and texture, and the low-level 
features do not match the keyword-based or content-based; 

presenting the first and second images to a user; and 

receiving feedback from the user as to whether the first and second images are 
relevant to the query further comprising: 

assigning a large weight to an association between the query keyword and the 
images deemed relevant by the user and/or assigning a small weight to an association 
between the query keyword and the example image." 



% 
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Please delete claim [32], replace with "32. An image retrieval system comprising: 
a query handler to handle both keyword-based queries having one or more 

search keywords and content-based queries having one or more low-level features of 

an image; 

a feature and semantic matcher to identify at least one of (1 ) first images having 
keywords that match the search keywords from a keyword-based query, and (2) second 
images having low-level features similar to the low-level features of a content-based 
query based on the first images, wherein the low-level features comprise color, shape 
and texture and the low-level features do not match the keyword-based or the content- 
based; 

a user interface to present the images identified by the feature and semantic 
matcher to a user, the user interface allowing the user to indicate whether the images 
are relevant to the query; and 

a feedback analyzer to train the image retrieval system based on user feedback 
as to relevancy, wherein the feedback analyzer to assign a large weight to an 
association between the query keyword and the images deemed relevant by the user 
and/or feedback analyzer to assign a small weight to an association between the query 
keyword and the example image and/or a feedback analyzer to assign a small weight 
to an association between the search keywords and the example image." 

Please delete claim [39], replace with "39. A computer-readable medium having 
computer-executable instmctions that, when executed, directs a computer to: 
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find a first set of images using both semantic-based image retrieval, and a second set of 
images using low-level feature-based image retrieval based on the first set of images, 
wherein the low-level features comprise color, shape and texture and the low-level 
features do not match the sematic-based or feature-based; 

present the images to a user so that the user can indicate whether the images 
are relevant; and 

concurrently conduct semantic-based relevance feedback and low-level feature- 
based relevance feedback, 

wherein the receiving feedback from the user as to whether the first and second 
images are relevant to the query further comprising: 

assigning a large weight to an association between the query keyword and the images 
deemed relevant by the user and/or assigning a small weight to an association between 
the query keyword and the example image." 



3. Claims 2-1 5, 20, 22-23. 28-35, 37 and 39-42 are allowed over prior art made of 



The following is an examiner's statement of reasons for allowance: None of the non 
prior art alone or incombination neither teaches nor suggest "initiating a search for 
images based on at least one query keyword in a query; identifying, during the search, 
first images having associated keywords that match the query keyword, and second 
images that contain low-level features similar to those of the first images identified as 
matching the query keyword, wherein the low-levels features comprise color, shape, 



Allowable Subject Matter 



record. 
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and texture and the low-level features do not match the search query; ranking the first 
and second images; presenting the first and second images to a user; and receiving 
feedback from the user as to whether the first and second images are relevant to the 
query further comprising: assigning a large weight to an association between the query 
keyword and the images deemed relevant by the user and/or assigning a small weight 
to an association between the query keyword and the example image." 

Any comments considered necessary by applicant must be submitted no later 
than the payment of the issue fee and, to avoid processing delays, should preferably 
accompany the issue fee. Such submissions should be clearly labeled "Comments on 
Statement of Reasons for Allowance." 



4. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. 

Aggarwal et al. (US. Patent No. 6,728,706 62) Patent date: 06/03/2003 



Shevade et al., "an experiental annotation system". Year 2003, ISBA: 1-5081 13, 
page 91-98. 

Lai etal., "PBIR-MM: multimodal image retrieval and annotation". Year 2002, 
ISBN: 1-581 13-620-X, page 421^22. 



5. Any inquiry conceming this communication or earlier communications from the 
examiner should be directed to Baoquoc N. To whose telephone number is (703) 305- 



Conclusion 



Fox etal. (US. Patent No. 6,574,632 62) 



Patent date: 03/03/2003 



Contact information 
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1949 or via e-mail BaoquocN.To@uspto.gov. The examiner can normally be reached 
on Monday-Friday: 8:00 AM - 4:30 PM, EST. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, John Breene can be reached at (703) 305-9790. 

Any inquiry of a general nature or relating to the status of this application or 
proceeding should be directed to the receptionist whose telephone number is (703) 305- 
3900. 

Any response to this action should be mailed to: 



The fax numbers for the organization where this application or proceeding is 
assigned are as follow: 

(703) 872-9306 [Official Communication] 

Hand-delivered responses should be brought to: 
Crystal Park II 
2121 Crystal Drive 
Arlington, VA 22202 
Fourth Floor (Receptionist). 

Baoquoc N. To 

May 13, 2004 /^^.^y. 



Commissioner of Patents and Trademarks 



Washington, D.C. 20231. 




